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Abstract 

Peer reviewed manuscripts contain a certain level of merit, as they 
have survived the scrutiny of reviewers who possess some expertise in 
the relevant area. This article discusses the purpose of reviewing 
manuscripts for publication in scholarly journals. Various aspects and 
issues of the peer review process are described, including reviewer 
responsibilities. Criticisms and concerns associated with the current 
peer review system are addressed, as well as guidelines for reviewing 
manuscripts. Finally, recommendations for improving the peer review 
process are provided. 


Introduction 

The publication of research and scholarly papers plays an integral role in advancing our 
understanding of and ability to act on issues related to workforce preparation and career 
development. At its very foundation, scholarly publication, regardless of field, rests on 
the editorial and peer review system. It is “. . . an inevitable part of the process of getting 
published” (Hawkey, 2001, p. 65). 

The process of soliciting peers (experts) to evaluate scholarly work prior to publication 
was initiated by Henry Oldenburg, editor of the first scientific journal Philosophical 
Transactions, as a means of providing a method for reporting experimental results that 
encouraged submissions while lessening the chances of one’s work being stolen 
(Berkenkotter, 1995). Despite its tenuous beginnings, and while not without critics or 
limitations, the review process today is intended to nurture “. . . a communal trust in the 
publication decision by creating a unique formal consultation among authors, editors, and 
reviewers, or ‘referees’, about the merits, scope, style, methods, substance, and 
knowledge claims of a potential article” (Chubin & Hackett, 1990, p. 84). The idea of 
nurturing trust and collegiality through peer review was reinforced by Young (2003) who 
suggests that the primary aims of peer review are to select/reject manuscripts for 
publication and make the entire process transparent, accurate, and practical. Benos, Kirk, 
and Hall (2003) offer even more specific direction. 
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The purpose of peer review is to ensure 1) quality, checking that no 
mistakes in procedure or logic have been made; 2) that the results 
presented support the conclusion drawn; 3) that no errors in citations to 
previous work have been made; 4) that all human and animal protocols 
conducted follow proper review and approval by appropriate 
institutional review committees; and, very importantly, 5) that the work 
is original and significant, (p. 48) 

Typically, the peer review process used by scholarly journals is fairly uniform. On 
receipt, editors first determine the overall quality and general suitability of the 
manuscript. If judged to be a poor fit with the journal or representing “sloppy, shoddy 
work,” the manuscript is rejected without review and returned to the author (Uchiyamam, 
Simnone, & Borko, 1999). While some argue that this action injects an unwanted element 
of bias and unbalanced control, the review process necessarily represents both objective 
and subjective elements (Coelho & LaForge, 2000). Murray and Raths (1996) supported 
the need for fairness and balance (what they called democratic principles) in the review 
process, and acknowledged that “. . . scholarship is undoubtedly enhanced by democratic 
principles.” Even so, they were quick to point out “. . . scholarly work is not determined 
by majority vote, nor are editorial judgments simply a matter of tallying votes” (p. 419). 

Another subjective aspect of the peer review process is the selection of reviewers. 
Typically, the editor selects three (or more) professionals based on some combination of 
the following qualities: familiarity with the topic, diversity, skill with the review process, 
sensitivity, honesty, and timeliness While reviewing a manuscript is work, Benos et al. 
(2003) consider the task “. . . an honor, not only because you are being recognized for 
your eminence in a particular area of research but also because of the responsibility and 
service you provide to the journal and the scientific community” (p. 47). 

Obviously, reviewers should be knowledgeable about the topic and have a clear 
understanding of the historical context in which the work was done (Benos et al., 2003). 
Sometimes a diverse group of reviewers is purposefully sought to gather opinions from 
various, perhaps even opposing, perspectives. This approach has advantages, particularly 
when critiquing qualitative or mixed methods designs. Flowever, “it can wreak havoc for 
the writer seeking to publish who happens to get reviewers who have little regard for (or 
outright hostility to) the writer’s approach, methods, or theoretical/disciplinary 
framework” (Berkenkotter, 1995, p. 246). For the most part, however, a diverse mix of 
opinions and perspectives is healthy and provides for rigorous and constructive critiques. 
Further, assembling a like-minded group of scholars does not guarantee comparable 
conclusions. In fact, it is rare to have a manuscript unanimously recommended for 
publication without revision (Murray & Raths, 1996). 

The importance of peer review should not be underestimated. In fact, the results of this 
process, i.e., published manuscripts, serve as a public record of our field in terms of 
interests, emphasis and contributions to workforce preparation, and the formation and 
evolution of discipline and professionalization (Benos et al., 2003; Uchiyamam et al., 
1999; Young, 2003). Further, peer reviewers and editors are often described as 
gatekeepers who monitor and construct the type and quality of new knowledge entering 
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the field and, perhaps, in the process preventing competing ideas from being published 
(Berkenkotter, 1995; Taylor, Beck, & Ainsworth, 2001). Yet, despite its importance, a 
relatively limited body of literature exists on peer review, including the specific criteria 
used to determine which manuscripts are selected for publication (Coelho & LaForge, 
2000; Rowney & Zenisek, 1980). What we do know is that many professionals lack an 
adequate working knowledge of how to effectively critique research, particularly novice 
scholars (Seals & Tanaka, 2000). Unfortunately, formal training in effective analysis of 
research manuscripts is rarely, if ever, available (Benos et al.). To address this concern, 
the remainder of this article focuses on select aspects of the editorial review process. We 
will, in turn, identify primary criticisms and concerns about the current system, offer 
guidelines for reviewing scholarly manuscripts, and give recommendations for improving 
the peer review system in use in the vast majority of scholarly journals, including the 
Journal of Career and Technical Education. 

Criticisms 

Criticisms of the peer review process are fairly easy to find in the literature and are often 
quite harsh. Take this assessment of peer review made by Ponsi (2003) as an example. 

Despite its wide acceptance, peer review has been subjected to a variety 
of criticisms: the evaluation procedures are often inadequately 
performed, and in general it can be said that research on the peer 
review process does not provide unquestionable evidence of its value. 

This time-consuming and resource-intensive process is slow, 
expensive, profligate of academic time, highly subjective, prone to bias, 
easily abused, poor at detecting gross defects, and almost useless for 
detecting fraud, (p. 444) 

Ouch! We wonder what Ponsi really thinks about the process. But wait, there’s more. A 
decade earlier, Bornstein (1990) decried the review process as “. . . unreliable, 
unconstructive, and biased in a number of ways (e.g., biased against nonsignificant 
findings, against replications of previous work, against unknown authors and less 
prestigious institutions, and against unpopular or counterintuitive findings)” (p. 672). 

Okay, if it’s really so bad, then why continue using it? Frankly, the answer comes down 
to this: Despite the flaws, we really don’t have anything better (Young, 2003). 
Berkenkotter (1995) agrees, but sees the review process more in terms of a social 
mechanism that allows a discipline’s experts to maintain quality control over new 
knowledge entering a field. “While the peer review is not infallible, it remains the 
primary means through which authority and authenticity are conferred upon scientific and 
scholarly papers by journal editors and the expert judges they have consulted” (p. 245). 
Arrington (1995) took a different perspective. 

If given all of this, a submission is recommended for publication by 
two or more blind reviewers, chances are it’s because it succeeds in 
spite of these complications and may stand a better chance of being 


©2004 - Journal of Career and Technical Education. Vol. 20, No. 2, Spring, 2004 - Page 43 



considered important and valuable by other interested and 
knowledgeable readers, (p. 252) 

In many ways, we concur with the observations of these authors. There is little doubt that 
the review process contains flaws. Surely, most everyone reading this article can attest to 
some unpleasant or frustrating experience incurred while trying to publish in a scholarly 
journal. However, we believe that the review process can work effectively and provides a 
fairly reliable mechanism for determining the quality of manuscripts for publication. 

One key to the success of the review process is the journal editor who must be vigilant 
about the subjective elements of the review process. Contrary to the belief of some, we 
don’t believe it is possible to eliminate subjectivity from the decision-making process. 
However, the editor can ensure that the overall process is fair. A first step in ensuring 
fairness is to identify criticisms of the process and then minimize the likelihood of their 
occurrence. 

Given the partially subjective nature of the review process, a number of criticisms have 
surfaced over the years. We address several of the more common ones are addressed in 
this section including a lack of understanding of reviewers’ (and editors’) proper role, the 
role of politics and conflict of interest, allowing emotion to enter into the decision¬ 
making process, and unequal treatment of various manuscript sections by reviewers. 
Although we focus on criticisms levied against the review process, we are encouraged by 
the optimism of Benos et al. (2003) who asserted that specific review techniques can be 
nurtured and learned. Perhaps understanding the problems is one of the first steps in that 
direction. 

Who’s Advocate? 

What is the role of manuscript reviewers? Should they be advocates for journals or for 
authors? The basic argument goes something like this: Reviewers are in a unique position 
to provide manuscript authors with guidance about unexplored aspects of their data 
analysis or recommendations for deeper interpretations of results. Acting as an advocate 
for the author, reviewers would find reasons for supporting the publication of the 
manuscript by providing “a critique that is positive, critical yet objective and balanced, 
contains no personally offensive comments, and is returned promptly. When specific 
criticisms are made, the reviewer should indicate precisely what the problems are and 
how they may be overcome” (Benos et al., 2003, p. 48). Sternberg (2003) asserts that any 
criticism, despite its severity, can be communicated in a tactful manner. 

Unfortunately, the experience of many authors is anything but collegial. In fact, authors 
often perceive an antagonistic relationship between themselves and reviewers, created by 
the general tone and content of reviewers’ comments. Fontaine (1995) suggests that the 
potential for a hostile, albeit undeclared, rivalry between authors and reviewers may be 
endemic to a system where both groups are basically vying for the same goal, recognition 
and publication of their work. As a result, many reviewers gradually adopt a critical, 
rather than collaborative, approach to the review process. This type of interaction is often 
the result of what Benos et al. (2003) refers to as adopting a journal advocacy role. In an 
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advocacy role for the journal, a reviewer’s job is to ensure that the best possible 
manuscripts appear in print. While acting as a journal advocate does not automatically 
result in harsh or disparaging treatment, the nature of the task (to serve in a gatekeeping 
role by providing a rigorous judgment of the quality, scholarship, and originality of a 
manuscript) and the system itself (double-blind review conducted by potential rivals) are 
potent catalysts for negative experiences to occur. 

Unequal Treatment of Manuscript 

When asked about the importance of various aspects of the review process, Coelho and 
LaForge’s (2000) journal reviewers considered 5 critical areas, including 
“. .. (a) accuracy of information; (b) reasonableness of conclusions based upon 
appropriate data analysis; (c) contribution the manuscript offers to the field; (d) 
readability, grammar, and style; and (e) organization in terms of logic (cogency) and 
standard format” (p. 6). Unfortunately, not all reviewers consider each of these areas in 
equal measure. In fact, it is common for reviewers, particularly novice ones, to 
“. . . overemphasize certain limitations (usually methodological), while missing other key 
points related to the scientific method that should be weighted more heavily” (Seals & 
Tanaka, 2000, p. 52). Even so, we don’t necessarily see this as a big problem in obtaining 
quality reviews. In fact, a savvy and knowledgeable editor can actually use this situation 
to her or his advantage by selecting reviewers known to have certain predispositions. A 
problem does exist, however, if reviewers recommend rejecting manuscripts because of 
an inability to separate major conceptual or methodological issues from routine editing 
concerns. To avoid this type of situation. Seals and Tanaka (2000) offer this advice, 

It is important for reviewers to organize their comments in a way that 
distinguishes between the major concerns on which the acceptability of 
the manuscript depends and the necessary ‘housekeeping’ chores 
associated with the revision process (correcting typographical errors, 
minor wording changes, adding informational details, etc.). The latter 
are indeed essential (they should not be considered ‘optional’), but 
author and editor both benefit from a clear delineation of a major and 
minor concerns of the reviewer, (p. 57) 

Sometimes reviewers will focus on minutiae while paying relatively little attention to 
necessary components of a manuscript. Hyman (1995) describes a situation where he 
engaged a colleague in a lengthy rebuttal about the appropriateness of using one 
particular formula versus another. In actuality, either formula was acceptable; the 
argument was about preference. By focusing on the detail, the potential contribution of 
the manuscript as a whole was lost. By bringing a sense of balance to the review process, 
a more equal treatment of the entire manuscript can be accomplished. 

Competing Recommendations (Feedback) 

While uneven treatment of a manuscript can be a bit frustrating, it is downright 
infuriating when three reviewers offer three different recommendations for a single 
manuscript that range from acceptance to rejection. What gives? How can three 
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supposedly intelligent people draw such diverse conclusions about the same manuscript? 
Actually, it shouldn’t be all that “surprising that different readers expect, notice, and 
value different qualities in an academic submission, a fact that explains why reviewers 
may offer conflicting reviews and verdicts” (Arrington, 1995, p. 251). When this occurs, 
and it occurs with relative frequency, the ultimate burden of deciding falls squarely on 
the shoulders of the journal editor who must determine which review(s) are more 
convincing (Berkenkotter, 1995). When decisions are split, reviews need to provide 
substantive feedback both in terms of problems and potential solutions. 

Emotion, Casting Aspersions, Politics, and Conflicts of Interest 

The literature is replete with stories of reviews containing ad hominem attacks and 
emotional language, accusations of political agendas, and conflicts of interest affecting 
the outcome of manuscript reviews. For example, over 70% of Fontaine’s (1995) 
respondents reported an experience where reviewers’ personal or professional agendas 
blatantly interfered with receiving an unbiased manuscript review. In our view, and that 
of Young (2003), unjustified reviewer biases have no place in the peer review process. 

Sternberg (2003) stated, “Many professionals write hostile reviews that at times have 
more of the characteristic of a personal attack than of a constructive critique of the work 
they are reviewing” (p. 159). It is troubling to hear of personal attacks at authors through 
written reviews. This concern is further heightened when considering the anonymous 
nature of the blind review process. True, it may be a bit difficult to keep emotions in 
check when reacting to the work of a colleague who shares equal passion for your topic, 
but it is our strong opinion that this form of review is never justified. In fact, Flyman 
(1995) indicates that not only should personal attacks be avoided, but great pains should 
be taken “. .. to avoid even the possibility that comments might be wrongly construed as 
a personal attack” (p. 181). Fie further advises us to (a) assume that authors are rational 
and capable people, (b) use leniency in judging others’ work, and (c) look for apparent 
weaknesses in how arguments are expressed rather than in the argument itself. “Ideally, a 
good [review] should allow the facts and arguments to speak for themselves. If you have 
a strong case, then you should not have to buttress it with valuative judgment” (p. 181). 
Additionally, authors are more likely to consider reviewers’ comments if criticism is 
constructive and offered in a neutral or supportive tone. 

Conflicts of interest can occur when reviewers have a vested stake in the outcome of the 
review process either as an author or as a potential rival. Houlihan, Hofschulte, Sachau, 
and Patten (1992) examined the potential conflict of interest when journal editors, 
associate editors, and editorial board members submit manuscripts to their own journals. 
They reported that approximately two-thirds of their sample had published in their own 
journals while serving in some editorial capacity, and over 80% felt that it was okay to do 
so provided that manuscripts went through blind review and, in the case of editors and 
associate editors, that they were handled by a guest editor. The good news is that it 
appears most editors and editorial boards are responsible in their handling of manuscripts 
submitted by scholars serving the journal. 
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Finally, we would be remiss if we did not say a few words about politics. Berkenkotter 
(1995) notes that it could be fairly easy for unscrupulous reviewers to use their position 
to act as gatekeepers, advancing the work of allies while preventing their competitors 
from getting published. In addition, instances have occurred where political pressure to 
shape educational publication has been wielded from forces outside the academy. Murray 
and Raths (1996) cited several concerns about the editorial process including the 
willingness of reviewers to do the work in a timely fashion, uninformed reviews, 
ideological or political bias, ethical conflicts of interest, and mistaken views of fairness 
held by the field. While naive to think we can eliminate all influence of politics, unethical 
practice, or conflicts of interest, it is possible to maintain an open and transparent review 
process. 


Manuscript Review 

Manuscript review is an intellectual process with both objective and subjective elements 
(Coelho & La Forge, 2000). Examination of the criticisms of the process provides some 
direction when considering how to review a manuscript. This section suggests both 
general reviewer responsibilities, as well as specific criteria they can use to review 
qualitative or quantitative research. 

General Reviewer Responsibilities 

Peer reviewers are generally charged with providing feedback to a journal editor about 
the merits of a particular scholarly work in terms of quality and contribution to the field. 
Ultimately, this feedback is used to decide about whether or not to publish the 
manuscript. So, what type of feedback is useful to editors? What constitutes the general 
responsibilities of peer reviewers? 

While the specific charge to reviewers may vary, some general responsibilities for peer 
reviewers have been identified. These include to (a) evaluate a manuscript’s strengths and 
weaknesses, as well as the subject matter and overall quality of the work, (b) report 
suspected duplicate publication fraud, plagiarism, or ethical concerns, (c) assist the editor 
in making a publication decision, (d) provide suggestions for improving the manuscript 
prior to publication, and (e) maintain the standards of the journal and discipline 
(Patterson & Bailar, 1985; Taylor et al., 2001). In performing their duties, reviewers are 
ethically bound to (a) review only those manuscripts for which they have expertise, (b) 
maintain confidentiality, and (c) write reviews in a timely, collegial, and constructive 
manner (Benos et al., 2003; Uchiyamam et al., 1999; Young, 2003). 

Issues In Manuscript Review 


General Issues 

Murray and Raths (1996) provide seven general criteria used to assess scholarly 
manuscripts including “(a) quality of the literature reviewed, (b) significance of the topic, 
(c) potential impact of the [research] on research and practice, (d) contribution to the 
field, (e) appropriate length, (f) clarity of expression, and (g) balance and fairness” 
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(p. 417). Others have also voiced their opinions about the essential criteria needed for a 
quality manuscript review mostly indicating a need to examine facts, interpretations, 
conclusions, scholarly and professional significance, and readability (Arrington, 2003). 
The essential question is, of course, the quality of the manuscript and its particular 
strengths and weaknesses. 

Section-Specific Issues in Manuscript Review 

Uchiyamam et al. (1999) acknowledge that editors and reviewers are interested in many 
aspects of a manuscript, although not always the same ones. While varied, common 
elements of manuscript reviews include the introduction (including rationale or context 
and purpose), method, results, and discussion (implications and significance of 
manuscript contents); what Seals and Tanaka (2000) refer to as the IMRAD format. Most 
importantly, reviewers must determine the overall quality and unique contribution of the 
work to the field at large. These general themes apply to both quantitative and qualitative 
research manuscripts (Taylor et al., 2001). The Appendix provides a list of general 
guidelines to consider when reviewing manuscripts. 

Introduction. The introduction provides a general context for the paper and a rationale 
for engaging in the work. This section provides the foundation for understanding the 
remainder of the article and establishes the importance or significance of the topic. The 
particular niche for this work is identified. Variables, concepts, ideas, and terms are 
introduced and defined. Specific connections between variables, people, or concepts are 
also described. At the conclusion of this initial section, the purpose of the study or paper, 
and specific questions or objectives are stated clearly. 

Eichorn and VandenBos (1985) observed that “. . . the literature research in a journal 
article cannot be exhaustive; rather it should selectively but systematically inform the 
reader about key theoretical and research issues and about previous and current research” 
(p. 1315). So, while relevant literature is cited in the introduction it should not constitute 
an exhaustive historical review nor be placed in a separate Review of Literature section. 
Rather, only literature providing an appropriate history and recognizing prior work 
leading to the present effort are included (American Psychological Association, 2001; 
Moore Burnett, & Moore, 1986; Seals & Tanaka, 2000; Uchiyamam et al., 1999). The 
introduction should lead readers to a logical end, i.e., the purpose of the paper, and 
clearly tie the work to a larger body of literature and an established theoretical base. 

Method. In research papers, the method section provides a detailed account of activities 
and technical issues addressed in the completion of the study. The method section 
provides evidence that the research is valid, reliable, and objective (Moore et al., 1986). 
Major subsections usually include some combination of the following: research design, 
population/sample/participants, means of data collection, researcher bias, experimental 
design and protocols (when applicable), procedure, and data analysis. For each of the 
sections, authors should assess and discuss relative strengths and weaknesses. 

In the case of quantitative research, a detailed description of the sample is necessary to 
understand the population being represented and the extent of generalizing results (Seals 
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& Tanaka, 2000). We recommend that evidence be provided to demonstrate the 
representativeness of the sample to the larger population, regardless of whether it is real 
or abstract. This can be fairly easy to accomplish if a thorough description of participants 
is given and compared to known information about the population. The need for this 
extra step seems especially important when nonrandom samples are used. In contrast, 
qualitative researchers are generally expected to provide detailed descriptions of 
participants, the specific sampling strategies employed, and the context from which they 
were identified. 

The description of quantitative instalments (and experimental treatments, when 
applicable) are crucial to readers’ understanding and should include (a) physical and 
conceptual descriptions of instalments and the constructs they are designed to measure, 
and (b) past evidence of validity and reliability, as well as a discussion about how validity 
was (re-)established for the present sample. When questionnaires are specifically 
designed for the research, the method used to develop and pilot test instalment items 
should also be addressed. In qualitative studies, researchers must describe the data 
collection protocols selected, depending on the nature of the strategy(ies) they used (e.g., 
semi-staictured interviews, document retrieval, and analysis). Typically, the method 
section also contains a brief account of major procedures used to complete the study. 

Results. The results section presents a summary of the research findings (Rudner & 
Schafer, 1999). Generally, this section is the longest one of a research manuscript, and 
contains both narrative and statistical information. It is important to note that while text 
and statistics complement each other, both should be able to stand alone. “The purpose of 
presenting both types of information is to make it easier for the audience to understand” 
(p. 12). In addition, findings presented in the results section should directly relate to the 
general problem and specific research objectives or questions under investigation (Moore 
et al„ 1986). 

Data results must be complete and presented in a clear and well-organized manner. In 
quantitative studies, all descriptive and, if appropriate, inferential statistics must be 
accurate. Tables and figures should be appropriately labeled with correct units and 
limited to only those that help to further the readers understanding of the data (Rudner & 
Schafer, 1999; Seals & Tanaka, 2000). Moore et al. (1986) stressed the importance of 
carefully interpreting results of a study, as incorrect or misinterpreted results could 
jeopardize the entire research effort. Complications encountered during research such as 
missing data, outliers, or miscoded data are also addressed in the results section. Finally, 
authors of quantitative research studies should address emerging concerns about the 
practical importance of their findings through analyses such as effect size and confidence 
intervals. 

Discussion and conclusions. In the final section of a research manuscript, results are 
discussed and interpreted. Conclusions are then drawn to establish the best possible 
answer to the research question(s) given the findings. Two errors commonly committed 
when writing conclusions are confusing results and conclusions, and overgeneralizing 
results (Moore et ah, 1986; Rudner and Schafer, 1999). It is also crucial that the 
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conclusions of a study are consistent with the results and firmly supported by the 
empirical evidence presented (Moore et al.; Seals & Tanaka, 2000). 

Many researchers begin their discussion by highlighting key results and reminding 
readers of the specific problem under investigation. Findings should then be related to 
previous research (Rudner & Schafer, 1999). Final conclusions and interpretations of data 
should be reasonable. In addition, the unique contribution of the study to our 
understanding of the selected topic and the implications of findings to both research and 
practice should be clearly delineated (Moore et al., 1986; Seals & Tanaka, 2000). 

Summary 

Effectively critiquing manuscripts is an essential element of the peer review process. A 
published manuscript has survived the scrutiny of reviewers who possess some expertise 
in that relevant area, thus conferring “added value” to the manuscript. (Benos et al., 2003; 
Hyman, 1995; Seals & Tanaka, 2000). Despite the criticisms and flaws associated with 
the peer review process, it remains our best option for judging the merits of scientific 
research. Thus, reviewers must strive to make the process as valid as possible, for “the 
review process can never be better than the reviewers who participate” (Meier, 1997, p. 
563). 

While reviewing is a privilege and responsibility, it is important to remember that it is a 
fallible process (Benos et al., 2003; Rudner & Schafer, 1999). A journal editor 
thoughtfully considers the feedback received from reviewers and makes a decision about 
the acceptability of the manuscript for journal publication. In the end, quality reviews 
will be rewarded by the success of the journal (Young, 2003). Arrington (1995) 
concludes, “For good or ill, reviewers must make recommendations to the editor about 
the quality of submissions that may see print, and what they recommend, and how they 
will have a direct bearing on the profession, the particular journal, and the reviewers 
themselves” (p. 250). 
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Appendix 

General Guidelines to Consider When Critiquing Manuscripts 


Introduction 


Is what is known or unknown about a topic succinctly stated? 

Is significance of the topic (research) established? 

Does the context or introduction lead to a logical end, i.e., the 
purpose, and clearly tie the investigation to a larger body of 
literature and an established conceptual or theoretical base? 

Is purpose of paper (research) clearly described with questions or 
objectives listed to guide reader? 

Are previous studies on the same topic cited, including their 
strengths and delimitations? _ 
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Method 


Results 


Discussion 


Are specific subsections identified with headings? 

Are methods described in sufficient detail for replication? 

Is the research design described along with accompanying 
advantages and delimitations? 

Is population and sample (for quantitative orientation) or group of 
participants and context (for qualitative orientation) described 
adequately (including approach used and resultant implications) 
and appropriate for question(s) posed? 

Are both the population and sample described so that some 
reasonable determination can be made regarding the 
representativeness of the sample to the intended population? 

Are instillments or other data collection procedures appropriate, 
valid, and reliable? 

For experimental studies, is the treatment described in sufficient 
detail and appropriate, valid, and reliable? 

Are data analyzed appropriately? 

Are the statistical techniques used appropriate for the research 
design? 

Is the potential of researcher bias addressed? _ 

Are findings presented in a well-organized, concise, and 
straightforward manner? 

If qualitative, are themes identified and described in detail? 
Are contradictory findings discussed? 

If quantitative, are both descriptive and inferential statistics 
(if appropriate) interpreted correctly and described in detail 
in the findings section? 

What is the practical importance of findings as evidenced by 
effect size and confidence intervals? 

Are the data results on all measurements included in the 
methods section described? 

Do the data appear reasonable? 

Are tables and figures effectively used to present findings? 
Have tables, figures, and narrative been used appropriately 

and effectively? _ 

Does discussion of results go beyond merely restating findings? 

Are conclusions and recommendations for practice based on 
research findings? 

Do the major findings of the study contribute to the field? 

Are experimental limitations of the study addressed? 

Is the significance of the results presented? 

Are implications of the findings and recommendations for future 
research discussed? 

Does the author(s) support statements with appropriate references? 
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